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CONVERGENCE  OF  MESH  ADAPTIVE  DIRECT  SEARCH 
TO  SECOND-ORDER  STATIONARY  POINTS  * 

MARK  A.  ABRAMSON  +  AND  CHARLES  AUDET  * 

Abstract.  A  previous  analysis  of  second-order  behavior  of  generalized  pattern  search  algorithms  for  unconstrained  and  linearly 
constrained  minimization  is  extended  to  the  more  general  class  of  mesh  adaptive  direct  search  (MADS)  algorithms  for  general  con¬ 
strained  optimization.  Because  of  the  ability  of  MADS  to  generate  an  asymptotically  dense  set  of  search  directions,  we  are  able 
to  establish  reasonable  conditions  under  which  a  subsequence  of  MADS  iterates  converges  to  a  limit  point  satisfying  second-order 
necessary  or  sufficient  optimality  conditions  for  general  set-constrained  optimization  problems. 

Key  words,  nonlinear  programming,  mesh  adaptive  direct  search,  derivative-free  optimization,  convergence  analysis,  second-order 
optimality  conditions 


1.  Introduction.  In  this  paper,  we  consider  the  class  of  derivative-free  mesh  adaptive  direct  search 
(MADS)  algorithms  applied  to  general  constrained  optimization  problems  of  the  form. 


min/(x), 


0.1) 


with  /  :  R"  — >  M  U  {+<*>}  and  flCI". 

We  treat  the  constraints  by  the  “barrier”  approach  of  applying  the  algorithm,  not  to  /,  but  to  the  barrier 
objective  function  /n  =  /  +  Vn,  where  \)/_o  is  the  indicator  function  for  £1;  i.e.,  it  is  zero  on  Cl,  and  infinity 
elsewhere.  If  a  point  x  is  not  in  Cl,  then  we  set  /_q  (x)  =  °°,  and  /  is  not  evaluated.  This  is  important  in  many 
practical  engineering  problems  in  which  /  is  expensive  to  evaluate. 

The  class  of  MADS  algorithms  was  introduced  and  analyzed  in  a  as  an  extension  of  generalized 
pattern  search  (GPS)  algorithms  BED  for  solving  nonlinearly  constrained  problems.  Rather  than  applying 
a  penalty  function  ED  or  filter  |5j  approach  to  handle  the  nonlinear  constraints,  MADS  defines  an  additional 
parameter  that  enables  the  algorithm  to  perform  an  exploration  of  the  space  of  variables  in  an  asymptotically 
dense  set  of  directions.  Under  mild  assumptions,  the  Clarke  calculus  together  with  three  types  of  tangent 
cones  (hypertangent,  Clarke  tangent  and  contingent  cones)  are  used  to  prove  convergence  of  a  subsequence 
of  iterates  to  a  point  satisfying  certain  first-order  conditions  for  optimality.  An  implementable  instance  of 
MADS  is  introduced  in  ||4],  in  which  positive  spanning  directions  are  chosen  in  a  random  fashion  and  almost 
sure  convergence  to  a  first-order  stationary  point  is  obtained.  A  similar  first-order  analysis  is  done  in  |fl5l 
for  the  DIRECT  algorithm. 

This  paper  extends  the  MADS  analysis  to  show  convergence  to  points  satisfying  certain  second-order 
stationarity  properties,  in  a  manner  similar  to  that  of  E  for  GPS .  An  important  result  of  E,  is  that  the  iterates 
produced  by  a  GPS  algorithm  on  a  sufficiently  smooth  problem  cannot  converge  in  an  infinite  number  of 
steps  to  a  local  maximizer.  We  show  here  that  it  may,  unfortunately,  converge  in  an  infinite  number  of  steps 
to  a  saddle  point.  The  analysis  in  the  present  paper  gives  sufficient  conditions  under  which  a  subsequence  of 
the  iterates  produced  by  a  MADS  algorithm  converges  to  a  strict  local  minimizer.  The  necessary  optimality 
condition  is  not  based  on  any  of  the  three  tangent  cones  used  in  a,  but  rather  on  the  cone  of  feasible 
directions. 
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The  paper  is  outlined  as  follows.  The  MADS  algorithm  is  briefly  described  in  Section[2]  with  first-order 
properties  restated  in  Section  [3]  Section  [4]  introduces  the  generalized  Hessian  lfl6l  with  some  associated 
properties,  followed  by  necessary  and  sufficient  second-order  optimality  conditions  and  convergence  results. 
Section  [5]  provides  some  examples  to  illustrate  the  strength  of  these  results,  and  Section  [6]  offers  some 
concluding  remarks. 

Notation.  R,  Z,  and  N  denote  the  set  of  real  numbers,  integers,  and  nonnegative  integers,  respectively. 
For  any  set  S,  int(.S')  denotes  its  interior,  and  cl(5')  its  closure.  For  any  matrix  A,  the  notation  a  £  A  means 
that  a  is  a  column  of  A.  For  x  £  R"  and  e  >  0,  we  denote  by  BE(x)  the  open  ball  {y  £  R"  :  ||y  —  x||  <  e}.  We 
say  that  /  is  C11  near  x  if  there  exists  an  open  set  S  containing  x  such  that  /  is  continuously  differentiable 
with  Lipschitz  derivatives  for  every  point  in  S.  The  reader  is  invited  to  consult  m  for  a  discussion  and 
examples  of  C11  functions. 

2.  Mesh  Adaptive  Direct  Search.  Like  GPS  methods,  each  iteration  k  of  a  MADS  algorithm  is  char¬ 
acterized  by  two  steps  -  an  optional  SEARCH  step  and  a  local  POLL  step,  in  which  fa  is  evaluated  at  specified 
points  that  lie  on  a  mesh.  The  mesh  is  constructed  from  a  finite  fixed  set  of  no  directions  D  C  R"  scaled 
by  a  mesh  size  parameter  A™  >  0.  The  directions  form  a  positive  spanning  set  m  (i.e.,  nonnegative  linear 
combinations  of  D  must  span  R"),  and  each  direction  d  £  D,  must  be  constructed  as  the  product  Gz,  where 
G  £  R"x"  is  a  nonsingular  generating  matrix,  and  z  £  Z"  is  a  vector  of  integers. 

The  following  definition,  taken  from  Q  and  Q,  precisely  defines  the  current  mesh  so  that  all  previously 
visited  points  lie  on  the  current  mesh. 

Definition  2.1.  At  iteration  k,  the  current  mesh  is  defined  to  be  the  following  union: 

Mk={J  {x  +  A'fDz  :  z  £  N"D}  , 

xeSk 

where  Sk  is  the  finite  set  of  points  where  the  objective  function  f  had  been  evaluated  by  the  start  of  iteration 
k,  and  So  is  a  finite  set  of  initial  feasible  points. 

In  both  the  SEARCH  and  POLL  steps,  the  algorithm  seeks  to  find  an  improved  mesh  point ;  i.e.,  a  point 
y  £  Mk  for  which  fa(y)  <  fa(xk),  where  xk  is  the  current  iterate  or  incumbent  best  iterate  found  thus  far. 

The  SEARCH  step  allows  evaluation  of  fa  at  any  finite  set  of  mesh  points.  Any  strategy  may  be  used, 
including  none.  This  is  more  restrictive  than  the  frame  methods  of  Coope  and  Price  m,  but  it  helps  to 
ensure  convergence  without  a  sufficient  decrease  condition  or  any  other  assumptions  on  mesh  directions. 
The  SEARCH  step  adds  nothing  to  the  convergence  theory,  but  well-chosen  SEARCH  strategies  can  greatly 
improve  algorithm  performance  (see  El  013  HD). 

In  the  POLL  step,  fa  is  evaluated  at  points  adjacent  to  the  current  iterate  in  a  subset  of  the  mesh  di¬ 
rections.  Unlike  GPS,  the  class  of  MADS  algorithms  has  a  second  mesh  parameter  Ap,  called  the  poll  size 
parameter,  which  satisfies  A’f  <  Ap  for  all  k.  and  also 

limA  1'  0  £>  limAf  =  0  for  any  infinite  subset  of  indices  K.  (2.1) 

ketc  keK  K 

Under  this  construction,  GPS  methods  now  become  the  specific  MADS  instance  in  which  Ak  =  Ap  =  A"‘. 

The  set  of  points  generated  in  the  POLL  step  is  called  a  frame,  with  Xk  referred  to  as  th t  frame  center. 
These  terms  are  now  formally  defined  as  follows: 

Definition  2.2.  At  iteration  k,  the  MADS  frame  is  defined  to  be  the  set: 

Pk  =  {xk  +  A'fd  '.  d  £  Dk}  C  Mk  , 
where  Dk  is  a  positive  spanning  set  such  that  for  each  d  £  Dk, 
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•  d  7^  0  can  be  written  as  a  nonnegative  integer  combination  of  the  directions  in  D  : 
d  =  Du  for  some  vector  u  G  N”°  that  may  depend  on  the  iteration  number  k 

•  the  distance  from  the  frame  center  Xk  to  a  poll  point  Xk  +  A'fd  is  bounded  by  a  constant  times  the 

poll  size  parameter  :  A™||d|  |  <  max{||of,||  :  d'  £  D} 

•  limits  (as  defined  in  Coope  and  Price  hll\D  of  the  normalized  sets  Dk  are  positive  spanning  sets. 

In  GPS,  the  set  of  directions  Dk  used  to  construct  the  frame  is  a  subset  of  the  finite  set  D.  There  is  more 

flexibility  in  MADS.  In  (4),  an  instance  of  MADS  is  presented  in  which  the  closure  of  the  cone  generated  by 
the  set  U”=i  |p]j  ;  d  G  At  j  equals  R”.  In  this  case,  we  say  that  the  set  of  poll  directions  is  asymptotically 
dense  in  R". _ 

Figure |2T~|illustrates  typical  GPS  and  MADS  frames  in  R2  using  the  standard  2 n  coordinate  directions. 
In  each  case,  the  mesh  Mk  is  the  set  of  points  at  the  intersections  of  the  horizontal  and  vertical  lines.  The 
thick  lines  delimit  the  points  that  are  at  a  relative  distance  equal  to  the  poll  size  parameter  A[’  from  the  frame 
center  Xk-  In  MADS,  the  mesh  size  parameter  A™  is  much  smaller  than  the  poll  size  parameter;  this  allows 
many  more  possibilities  in  the  frame  construction. 


FIG.  2.1.  GPS  and  MADS  frames  Pk  =  [p] .  jr .  jf .  p4 }  around  the  frame  center  Xk,  with  the  same  poll  size  parameter 

If  the  POLL  step  fails  to  produce  an  improved  mesh  point,  Pk  is  said  to  be  a  minimal  frame  with  minimal 
frame  center  Xk .  If  either  the  SEARCH  or  POLL  step  is  successful  in  finding  an  improved  mesh  point,  the 
improved  mesh  point  becomes  the  new  current  iterate  Xk+ 1  G  G,  and  the  mesh  is  either  retained  or  coarsened. 
If  neither  step  is  successful,  then  the  minimal  frame  center  is  retained  as  the  current  iterate  (i.e.,  Xk+ 1  =  Xk) 
and  the  mesh  is  refined. 

Rules  for  refining  and  coarsening  the  mesh  are  as  follows.  Given  a  fixed  rational  number  x  >  1  and  two 
integers  w~  <  1  and  w  >  0,  the  mesh  size  parameter  A™  is  updated  according  to  the  rule, 

At+t  =  xw*A“ 

f  {0, 1, . . .  ,w+}  if  an  improved  mesh  point  is  found 
for  some  w*  G  j  {w_)W_  +  1)  _  J}  otherwise 


p‘ 

r4 

xk 

p2 

P J 

A  MADS  frame 


P2 

P3' 

*  xk 

p' 

p 4 

A  GPS  frame 


(2.2) 
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The  class  of  MADS  algorithms  is  stated  formally  as  follows: 

A  GENERAL  MADS  ALGORITHM 

•  INITIALIZATION:  Let  xo  E  £2,  set  Aq  >  Aq  >  0.  Set  the  iteration  counter  k  to  0. 

•  SEARCH  AND  POLL  STEP:  Perform  the  SEARCH  and  possibly  the  POLL  steps  until  an  improved  mesh  point 
xa+i  is  found  on  the  mesh  M a  (see  Definition|2.1}. 

-  OPTIONAL  search:  Evaluate  fa  on  a  finite  subset  of  trial  points  on  the  mesh  M^. 

-  Local  POLL:  Evaluate  /jj  on  the  frame  Pf.  (see  Definition|2.2[>. 

•  PARAMETER  update:  Update  A™+1  according  to  \2.2\,  and  A(’+1  according  to  (|2. 1[>.  Increase  k  <—  k+  1 
and  go  back  to  the  SEARCH  and  POLL  step. 

3.  Existing  First-Order  Stationarity  Results.  Before  presenting  new  results,  we  reproduce  known 
convergence  properties  of  MADS,  originally  published  in  i).  All  results  are  based  on  the  following  as¬ 
sumptions: 

Al.  A  feasible  initial  point  xo  is  provided. 

A2.  The  initial  objective  function  value  /(xo)  is  finite. 

A3.  All  iterates  {xa }  generated  by  MADS  lie  in  a  compact  set. 

Under  these  assumptions,  Audet  and  Dennis  a  prove  that 

liminfA?  =  lim  inf  A/  =  0. 

k — >+oo  k — >+oo 

This  ensures  the  existence  of  infinitely  many  minimal  frame  centers,  since  A("  only  shrinks  when  a  minimal 
frame  is  found.  The  following  definition,  taken  from  m,  is  needed  for  later  results. 

DEFINITION  3.1.  A  subsequence  of  the  MADS  iterates  consisting  of  minimal  frame  centers,  Ixa/ask 
for  some  subset  of  indices  K,  is  said  to  be  a  refining  subsequence  if  converges  to  zero. 

Let  x  be  the  limit  of  a  convergent  refining  subsequence.  IflimkeL  pj/  exists  for  some  subset  L  C  K  with 
poll  direction  </  £  £/,  and  if  x&  +  Afd^  £  LI  for  infinitely  many  k  £  L,  then  this  limit  is  said  to  be  a  refining 
direction  for  Jc. 

Existence  of  refining  subsequences  for  MADS  was  proved  in  (4).  The  following  four  definitions  191(17], 
(20)  are  needed  in  the  main  theorems. 

Definition  3.2.  A  vector  v  £  R"  is  said  to  be  a  hypertangent  vector  to  the  set  LI  c  R"  at  the  point 
x  E  LI  if  there  exists  a  scalar  e  >  0  such  that 

y  +  twELl  for  ally  E  LIE] B£(x),  wEB£(v)  and  0  <  t  <  E.  (3.1) 

The  set  of  hypertangent  vectors  to  LI  at  x  is  called  the  hypertangent  cone  to  LI  at  x  and  is  denoted  by  Tq  (x). 

Definition  3.3.  A  vector  v  E  R"  is  said  to  be  a  Clarke  tangent  vector  to  the  set  LI  c  R"  at  the  point 
x  E  cl(£2)  if  for  every  sequence  {y^}  of  elements  of  LI  that  converges  to  x  and  for  every  sequence  of  positive 
real  numbers  {fy  }  converging  to  zero,  there  exists  a  sequence  of  vectors  {vva}  converging  to  v  such  that 
yk  +  tkWk  E  LI.  The  set  Tq  (x)  of  all  Clarke  tangent  vectors  to  LI  at  x  is  called  the  Clarke  tangent  cone  to  LI 
at  x. 

Definition  3.4.  A  vector  v  E  R"  is  said  to  be  a  tangent  vector  to  the  set  LI  c  R”  at  the  point  x  E  cl(£2) 
if  there  exists  a  sequence  {va}  of  elements  of  LI  that  converges  to  x  and  a  sequence  of  positive  real  numbers 
{A-a}  for  which  v  =  limAA-AGr  —  x).  The  set  T^°(x)  of  all  tangent  vectors  to  LI  at  x  is  called  the  contingent 
cone  (or  sequential  Bouligand  tangent  cone)  to  LI  at  x. 

Definition  3.5.  The  set  LI  is  said  to  be  regular  at x  if  T^  (x)  =  t£°(x).  In  addition  to  these  definitions, 
we  add  the  following  clarifying  remarks,  due  to  Clarke  0  unless  otherwise  noted: 
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Any  convex  set  is  regular  at  each  of  its  points, 
(x)  and  Tq 

M  C  Tg(x)  c  t£°(x). 


Both  7q° 
rrH 
1Q. 

Rockafellar 

rH( 


i  p_  (x)  are  closed. 


and  both  Tq  (x)  and  Tp  (x)  are  convex. 


showed  that,  if  Tff  (x)  is  nonempty,  Tff  (x)  =  int(7^,' (x)),  and  therefore,  Tp  (x)  = 

ci(r"(x)). 

In  order  to  establish  the  results  of  this  section,  we  apply  a  generalization  of  the  Clarke  (9J  directional 
derivative,  as  presented  in  DU,  in  which  function  evaluations  are  restricted  to  points  in  the  domain.  Specifi¬ 
cally,  the  Clarke  generalized  directional  derivative  of  the  locally  Lipschitz  function  /  at  x  G  £2  in  the  direction 
v  G  R"  is  defined  by 


/°(x;v)  := 


limsup 

y  — >  x,  y  €  LI 
t 1  0,  y  +  tv  G  LI 


f(y+tv)-f(y) 


(3.2) 


The  next  definition,  also  from  0,  provides  some  nonsmooth  terminology  for  stationarity. 

DEFINITION  3.6.  Let  f  be  Lipschitz  nearx  G  LI.  Then,  x  is  said  to  be  a  Clarke,  or  contingent  stationary 
point  of  f  over  LI,  if  f°  (x;  v)  >  0  for  every  direction  v  in  the  Clarke  tangent  cone,  or  contingent  cone,  to  D 
at  x,  respectively. 

In  addition,  x  is  said  to  be  a  Clarke,  or  contingent  KKT  stationary  point  of  f  over  LI,  if  — V/(x)  exists 
and  belongs  to  the  polar  of  the  Clarke  tangent  cone,  or  contingent  cone,  to  LI  at  x,  respectively. 


If  LI  =  W  or  x  lies  in  the  interior  of  LI,  then  a  stationary  point  as  described  by  Definition  3.6  meets  the 
condition  that  /°(x;v)  >  0  for  all  i>Gt".  This  is  equivalent  to  0  G  3/(x),  the  generalized  gradient  of  /  at 
which  is  defined  by 


3/(x)  :=  {s  G  M"  :  /°(x;  v)  >  vTs  for  all  v  G  R"  } . 


The  function  /  is  said  to  be  strictly  differentiable  at  x  if  the  generalized  gradient  of  /  at  x  is  a  singleton;  i.e., 

3/W  =  {V/WI- 

We  now  restate  the  main  results  from  0.  Theorem [3/7] is  a  directional  result  obtained  under  very  mild 
assumptions,  and  Theorem  3.8  the  main  result  of  this  section,  is  a  restatement  of  four  different  theorems 
found  in  0. 

THEOREM  3.7.  Let  f  be  Lipschitz  near  a  limit  x  G  LI  of  a  refining  subsequence,  and  v  G  T»(x)  be  a 
refining  direction  for  x.  Then  the  generalized  directional  derivative  of  f  atxin  the  direction  v  is  nonnegative, 
i.e.,/°(x;v)  >0. 

THEOREM  3.8.  Let  x  G  LI  be  the  limit  of  a  refining  subsequence,  and  assume  that  Tp{x)  ^  0  and  the 
set  of  refining  directions  is  dense  in  Tp[x). 

1.  If  f  is  Lipschitz  near  x,  then  x  is  a  Clarke  stationary  point  of  f  on  LI. 

2.  If  f  is  strictly  differentiable  at  x,  then  x  is  a  Clarke  KKT  stationary  point  of  f  on  LI. 

Furthermore,  if  LI  is  regular  at  x,  then  the  following  hold: 

1.  If  f  is  Lipschitz  near  x,  then  x  is  a  contingent  stationary  point  of  f  on  LI. 

2.  If  f  is  strictly  differentiable  at  x,  then  x  is  a  contingent  KKT  stationary  point  of  f  on  LI. 


4.  New  Second-Order  Stationarity  Results.  This  section  contains  second-order  convergence  theory 
for  MADS.  In  Section |?T|  we  recall  the  definition  of  the  generalized  Hessian  and  identifies  some  useful 
properties.  In  Section|4.2|we  present  second-order  necessary  and  sufficient  conditions  for  optimality  for  set- 
constrained  optimization  problems.  Finally,  in  Section|4~3|  we  establish  conditions  under  which  convergence 
of  MADS  iterates  to  a  point  satisfying  second-order  necessary  and  sufficient  conditions  is  achieved. 
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4.1.  Generalized  Second-Order  Derivatives.  Before  proving  convergence  to  second-order  points,  we 
present  nonsmooth  notions  of  second  derivatives  and  introduce  second-order  optimality  conditions.  Gen¬ 
eralized  second-order  directional  derivatives  are  developed  in  ifTOl  and  lfl6l.  consistent  with  the  Clarke  [0 
calculus  for  first  order  derivatives.  In  this  paper,  we  follow  the  Hirriart-Urruty  et  al.  EH  definition  of  a 
generalized  Hessian,  given  as  follows. 

Definition  4.1 .  Let  g  :  R”  — >■  R  be  C1,1  near  x  G  LIC  R".  The  generalized  Hessian  of  g  at  x,  denoted 
by  d2g(x),  is  the  set  of  matrices  defined  as  the  convex  hull  of  the  set 

{A  G  R”xn  :  there  exists  Xk  — >  x  with  g  twice  differentiable  at  Xk  and  V2g(xk)  — >  A} . 


By  construction,  d2g(x)  is  a  nonempty,  compact,  and  convex  set  of  symmetric  matrices  Uhl.  The 
function  g  is  said  to  be  twice  strictly  differentiable  at  x  if  the  generalized  Hessian  is  a  singleton;  i.e. ,  d2g(x)  = 
{V2g(x)}.  Furthermore,  as  a  set-valued  mapping,  x  =4  d2g(x)  has  two  key  properties,  also  identified  in  Q6), 
which  are  necessary  to  establish  optimality  conditions  in  the  next  section, 
is  a  locally  bounded  set-valued  mapping: 

Given  a  matrix  norm  ||  •  ||,  there  exists  an  e  >  0  and  T|  G  R  such  that 

sup{ ||A||  :  A  G  d2g(y),y  G  Be(a)}  <  r|; 
is  a  closed  set-valued  mapping: 

If  Xk  — >  x  and  A*  — » A  with  Ak  G  d2g(xk)  for  all  k,  then  A  G  d2g(x). 

The  following  second-order  Taylor  series  result  also  comes  from 

THEOREM  4.2.  Let  g  :  R”  — R  be  C1,1  in  a  open  set  U  C  R",  and  let  [a,b]  CU  be  a  line  segment.  Then 
there  exist  an  x  G}a,b[  and  a  matrix  Ax  G  d2  /(x)  such  that 

g(b)  =  g(a)  +  (b-a)TVf(a)  +  ^(b-a)TAx(b-a). 


In  the  next  section,  we  apply  this  result  to  feasible  points  that  may  lie  on  the  boundary  of  LI.  We  are  able  to 
do  this  because  our  assumptions  on  the  local  smoothness  off  are  independent  of  LI. 


4.2.  Second-Order  Optimality  Conditions.  Second-order  necessary  and  sufficient  optimality  condi¬ 
tions  for  constrained  problems  are  traditionally  expressed  in  terms  of  the  Lagrangian  function.  However,  our 
use  of  the  barrier  approach  in  handling  constraints  provides  no  useful  information  about  the  constraint  gradi¬ 
ents,  and  thus  prevents  us  from  proving  anything  with  respect  to  traditional  optimality  conditions.  Therefore, 
instead  of  dealing  with  the  Lagrangian  function,  we  extend  optimality  conditions  for  set-constrained  prob¬ 
lems  (see  (8)  for  further  discussions). 

We  now  establish  Clarke-based  second-order  necessary  and  sufficient  conditions  for  set-constrained 
optimality.  The  proof  for  the  former  is  very  similar  to  one  found  in  lfl6ll  for  unconstrained  problems,  the 
only  difference  being  the  first-order  condition  satisfied  by  the  local  minimizer.  It  is  expressed  in  terms  of 


feasible  directions,  formally  given  in  Definition  4.3 

Definition  4.3.  The  direction  v  G  R"  is  said  to  be  feasible  for  LI  C  R"  at  x  G  LI  if  there  exists  an  8  >  0 
for  which  x  +  tv  G  LI  for  all  0  <  t  <  £.  The  set  of  feasible  directions  for  LI  at  x  G  LI  is  a  cone  and  is  denoted 
by  Tq  (  x  ). 

It  follows  immediately  that  T^ (x)  C  t£ (x)  C  Tq°(x)  for  any  x  G  LI.  Moreover,  if  Tq  (x)  yL  0  for  some 
xGLI,  and  if  LI  is  regular  at  x,  then  cl(7^ (x))  =  cl(7^ (x))  =  T^1  (x)  =  Tq°(x).  However,  without  regularity 
it  is  possible  that  either  of  the  following  holds: 
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•  Tq  (x)  C  int(7^'  (x)):  e.g.,  if  LI  =  {(a,b)  £  R2  :  a  >  0  or  b  >  0},  then  T^{ 0,0)  =  R+  and  t£( 0,0)  = 

o. 

•  d(T£(x))  C  int(r^/(x)):  e.g.,  if  £2  =  M2  \  {(—  \,b)  £  R2  :  b  €  R,k  =  1,2, . . .},  then  Tq  (0,0)  = 
{(ci,b)  £  R2  :  a  >  0}  and  7^  (0,0)  =  R2. 

THEOREM  4.4  (Second-order  necessary  condition  for  set-constrained  optimality.).  Let  x*  €  LI  be  a 
local  solution  of  (1.1 1.  If  f  is  C1,1  near  x*,  then  any  feasible  direction  v  £  (x* )  for  which  vTVf(x*)  =  0 

satisfies  vT  Av  >  0  for  some  A  £  d2f[x*). 

Proof.  Let  v  £  R"  be  a  feasible  direction  that  satisfies  vT  V/(x*)  =  0,  and  consider  the  sequence  {xj.},  where 
Xk  =  x*  +  lv.  It  follows  that  Xk  £  LI  when  k  is  sufficiently  large.  Then  by  second-order  Taylor  series  in  a 
neighborhood  of  the  local  minimizer  x* ,  we  have  for  each  k  sufficiently  large 


0  <  f(xk)-f(x*)  =  ^Vf(x*)Tv+^vTAkv  =  vTAkV, 


(4.1) 


where  Ak  £  d2f(xk)  for  some  xk  £]x* 

Since  d2f  is  locally  bounded  and  xk  ~ >  x*,  the  sequence  {A*}  is  locally  bounded  and  thus  possesses  an 
accumulation  point  A.  Furthermore,  since  d2f  is  a  closed  set- valued  mapping,  we  have  A  £  ()2  f[x*).  Taking 
limits  in  (|4. 1  b  leads  to  vTAv  >0.  ■ 


Theorem  4.4  applies  to  the  set  of  hypertangent  vectors  as  well  as  feasible  directions  since  the  set  of 
feasible  directions  contains  the  hypertangent  cone.  However,  this  necessary  condition  does  not  necessarily 
hold  for  directions  in  the  Clarke  tangent  or  contingent  cone,  as  the  following  example  shows. 

Example  4.5.  Consider  the  quadratic  optimization  problem,  in  which  f  :  R2  — >  M  is  defined  by 
f{a1b)  =  —(a2  -Lb2),  and  LI  =  {( a,b )  £  M2  :  a2  +  (b—  l)2  <  1}.  The  optimal  solution  is  at  (0,2),  where 

Tq  (0,2)  =  To  (°,2)  =  {(vi,v2)  :  v2  <  0}  and  t£'( 0,2)  =  t£°(0,2)  =  {(vi,v2)  :  v2  <  0}. 

The  direction  v  =  (l,0)r  £  T^  (0,2)  =  Tq°{  0,2),  is  not  a  feasible  direction  and  makes  a  zero  inner  product 
with  V/( 0,2)  =  (0,  —f)T,  but  the  Hessian  matrix  is  given  by 


V2/(  0,2)  = 


—2  0 

0  —2 


which  yields  vrV2/( 0,2)v  =  —2  <  0. 

Theorem  4.6  (Second-order  sufficient  condition  for  set-constrained  optimality.).  Let  x*  £  LI  be  a 
contingent  stationary  point  for  the  optimization  problem  defined  in  (1.1),  and  suppose  that  Tq  (x*)  0  and 

that  LI  is  convex  near  x*.  If  f  is  C1,1  near  x* ,  and  ifvTAv  >  0  for  all  matrices  A  £  d2  f(x*)  and  all  nonzero 
tangent  directions  v  £  Tq°(x)  that  satisfy  vT  V/(x*)  =  0,  then  x*  is  a  strict  local  solution  of  (1.1 1. 

Proof.  The  proof  is  by  contraposition.  Suppose  that  x*  is  not  a  strict  local  minimizer.  Then  there  exists  a 
sequence  {y*}  C  £2  (with  v*  f=x*)  converging  tox*  satisfying  f{yk)  <  f(x*)  for  all  k.  By  taking  subsequences 


if  necessary,  we  can  assume  that  the  sequence  {wk}  with  Wk  = 


yt-x 
I  lyt-x* 


converges  to  some  vector  v  £  ' 


Local  convexity  of  LI  near  x*  implies  that  v  and  wk  are  contingent  directions  for  all  k  >  /:,  for  some 
integer  £  >  0.  Moreover,  since  x*  is  assumed  to  be  a  contingent  stationary  point,  and  since  /  is  continuously 
differentiable,  then  vrV/(x*)  >  0  and  wJV/(x*)  >  0  for  all  k>£.  However,  since  f(yk)  <  fix*)  for  all  k, 
then  vTVf(x*)  =  0. 

on  Taylor  series  ensures  that  for  each  k  >  £,  there  exists  some  matrix  Ak  £  d2  f(xk)  with 


Theorem 


4.2 
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xk  C]x*,yk[  such  that 


o  >  f(yk)-f(x*)  =  (yk-x*)  V/(x*)  +  -(y*-x*)  Ak(yk-x*) 
>  ^iyk-x*)1 Ak(yk-x*). 


(4.2) 


Now,  since  xk  — ♦  x*,  and  since  d2  f(x*)  is  a  closed  locally  bounded  set-valued  mapping,  then  there  exists  an 
accumulation  point  A  £  d2f(x*)  of  the  sequence  {Aa}.  Dividing  ( 4.2 )  by  \\yk  —  x*\\2  and  taking  limits  leads 
to  0  >  jVTAv,  where  vf^O  belongs  to  t£°(x*)  and  satisfies  v7V/(x*)  =  0.  ■ 

The  previous  theorem  requires  as  an  assumption  that  the  set  £2  is  locally  convex.  The  following  example 
shows  that  regularity  of  £2  is  not  sufficient  to  guarantee  a  local  minimizer. 

EXAMPLE  4.7.  Consider  the  quadratic  optimization  problem,  in  which  f  :  R2  — >  R  is  defined  by 
f(a,b)  =  a2  +  2  b,  and  £2  =  {( a,b )  £  R2  :  2  a2  +b>  0}.  The  solution  xT  =  (0,0)  is  a  contingent  stationary 
point,  and  £2  is  regular  at  x  since 


Tq{x)  =  Tq°(x)  =  {(vi,v2) :  v2  >  0}. 

The  vector,  v  =  (l,0)r  £  Tff (x)  satisfies  vTV f(x)  =  0  and  v/  V2/(x)v  =  2  >  0.  However,  (e,— e2)  belongs 
to  the  strict  interior  of  Cl  for  all  E  ^  0,  and  /(e,  — £2)  =  — E2  <  0  =  /(x). 

4.3.  Second-Order  Stationarity  Results  for  MADS.  The  next  two  results  are  the  main  contributions 
of  this  paper.  The  first  theorem  establishes  convergence  of  a  subsequence  of  MADS  iterates  to  a  point 
satisfying  the  second-order  necessary  condition  identified  in  Theorem  |4.4|  and  the  second  establishes  the 


sufficiency  conditions  of  Theorem  4.6 


THEOREM  4.8.  Let  f  be  C1,1  near  a  limit  x  of  a  refining  subsequence,  and  assume  that  T^{x)  ^  0  and 
that  £2  is  regular  near  x.  If  the  set  of  refining  directions  is  dense  in  Tq[x),  then  x  satisfies  the  second-order 
necessary  condition  for  set-constrained  optimality. 

Proof.  Let  v  £  R"  be  any  nonzero  feasible  direction  that  satisfies  i,y  V/  (x)  =  0,  and  suppose,  by  way  of 
contradiction,  that  vTAv  <  0  for  all  matrices  A  £  d2f(x).  Since  32/(x)  is  nonempty  and  compact,  and  d2 f 
is  a  closed  set-valued  mapping,  there  exists  some  E  >  0  such  that  vT Av  <  0  for  all  A  £  32/(x)  and  for  all 
x  £  Bz(x). 

Let  K  denote  the  set  of  indices  of  unsuccessful  iterations.  Regularity  of  £2,  together  with  the  assumption 
that  Tg(Jc)  0  guarantee  that  cl(7^(x))  =  T^° (x)  =  01(7^ (x)).  Therefore,  the  denseness  of  the  set  of 
refining  directions  in  T^{x)  ensures  the  existence  of  {wk}keK  converging  to  v  with  wk  =  dk  £  Dk  for 
each  k  £  K.  Taylor  series  yields 


f(xk  +  A [dk)  -  f(xk)  =  A pkdk  V/(xi)  +  -  (A l)  dk  A+ dk, 


f{xk  -  A  pkdk)~  f{xk)  =  -A  pkdlkVf{xk)  +  A  (Apk)2dlAk  dk 


(4.3) 

(4.4) 


where  At  £  32/(x+)  for  some  x+  G]xk,xk+Akdk[  and  Ak  £  32/(x  )  for  some  x  £]xjt,x*  —  A kdk[.  Since 
Ak  — >  0+  and x>  — » x,  there  is  a  subsequence  for  which  Ak  converges  to  some  A+  £  32/(x),  and  Ak  converges 
to  some  A'  £  32/(x).  Moreover,  since  a2/(x)  is  a  convex  set,  then  A  =  j  (A+  +  A  )  £  32/(x). 

Adding  (|4.3|  and  (4.4 1  and  substituting  dk=  ||t//:|| yields 


1 


A£  ||<4|| 


f(xk  +  Ap\\dk\\wk)  -f(xk) 

K\\M\ 


f(xk  —  Ak\\dk\\wk)  -f(xk) 
APk\\dk\\ 


=  wkAkwk , 


(4.5) 
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where  Ak  =  \  ( A £  +  Af).  Furthermore,  since  w k  — >  v  and  vTAv  <  0,  there  exists  y  <  0  such  that  w^Akwk  < 
y  <  0  for  all  sufficiently  large  k  £  K,  which  forces  the  left-hand  side  of  ( 4.5  i  to  also  be  negative  and  bounded 
away  from  zero.  But  since  dk  £  D k  for  all  sufficiently  large  k  £  K,  we  have  that  f(xk)  <  f{xk  +  A pkdk),  which 
makes  nonnegative  the  first  term  of  the  left-hand  side  of  (|4.5[>  (for  all  sufficiently  large  k  £  K ).  Thus  it  must 
be  the  case  that 


f(xk  -Apk\\dk\\ wk )  ~f(xk) 

a£||4|| 


(4.6) 


>  in  K  yields  V/(x)r(— v)  <  0,  or  V/(x)rv  > 


for  all  sufficiently  large  k£K.  Taking  the  limit  of  ( 4.6 1  as  k 
0,  which  contradicts  the  assumption  that  V/(x)rv  =  0.  ■ 

The  following  result  shows  that  the  sufficient  conditions  of  Theorem  |4.6|can  be  satisfied  by  a  subse¬ 
quence  of  MADS  iterates,  given  stronger  hypotheses  than  those  of  Theorem|4.8| 

THEOREM  4.9.  Let  f  be  twice  strictly  differentiable  at  a  limit  x  of  a  refining  subsequence,  and  assume 
that  Tg  (x)  jk  0;  LI  is  convex  near  x,  and  V2/(x)  is  nonsingular.  If  the  set  of  refining  directions  is  dense  in 
Tg  (x),  then  x  is  a  strict  local  minimizer  off  on  LI. 

Proof.  Since  /  is  twice  strictly  differentiable  at  x,  cf  f(x)  =  {V2/(x)}.  Thus,  it  follows  from  Theorem  4.8 
that  vrV2/(x)  v  >  0  for  all  feasible  directions  satisfying  V/(x)rv  =  0.  But  since  V2/(x)  is  assumed 

to  be  nonsingular,  this  inequality  is  strict.  Furthermore,  by  Theorem|3.8|and  the  smoothness  of  /  nearf,  x  is 
a  first-order  contingent  stationary  point.  Thus  the  hypotheses  of  Theorem  4.6  are  satisfied,  and  the  result  is 
proved.  ■ 

Clearly,  these  are  strong  results  for  a  direct  search  method.  However,  in  practice,  achieving  denseness 
of  the  refining  directions  in  the  hypertangent  cone  (a  key  assumption)  requires  increasingly  more  poll  di¬ 
rections  per  iteration.  To  overcome  this  problem,  an  implementable  instance  of  MADS  is  introduced  in  a, 
called  LTMADS,  in  which  the  positive  spanning  directions  used  at  each  iteration  are  limited  in  number,  but 
chosen  randomly  from  among  the  increasing  number  of  possible  poll  directions.  While  this  is  not  difficult 
to  implement,  the  drawback  is  that  denseness  of  the  refining  directions  is  only  achieved  almost  surely  (i.e., 
with  probability  one).  Thus,  in  practice,  the  convergence  results  proved  both  here  and  in  a  are  only  at¬ 
tained  almost  surely.  This  is  a  weaker  measure  of  convergence,  but  it  works  well  in  practice  a.  We  apply 
LTMADS  to  one  of  the  numerical  examples  in  the  next  section. 


5.  Examples.  Second  order  results  for  GPS  are  presented  in  JT].  They  are  not  as  strong  as  those 
presented  here  for  MADS.  In  this  section,  we  illustrate  this  difference  through  a  series  of  three  bound  con¬ 
strained  or  unconstrained  quadratic  examples  in  ]R2.  The  first  HI  illustrates  how  GPS,  but  not  MADS,  can 
converge  to  a  local  maximizer.  The  second  shows  how  GPS,  but  not  MADS,  converges  in  an  infinite  number 
of  iterations  to  a  saddle  point.  Finally,  since  the  second  example  requires  an  uncommon  set  of  parame¬ 
ter  choices,  we  recall  a  simpler  numerical  example  12  with  more  realistic  parameter  choices  to  show  how 
MADS  avoids  a  saddle  point  having  narrow  cones  of  descent,  which  GPS  with  reasonable  parameter  choices 
does  not. 


5.1.  An  example  where  GPS  stalls  at  a  global  maximizer  for  a  minimization  problem.  Consider 
the  unconstrained  problem  of  minimizing  the  function  f(a,b)  =  -(ah)2,  subject  to  —2  <  a.h  <  2,  with  a 
starting  point  at  (0,0),  the  global  maximizer.  Using  standard  coordinate  directions  and  their  negatives  as 
poll  directions,  GPS  will  stall  at  the  starting  point  without  moving.  This  is  consistent  with  the  result  proved 
in  12,  that  convergence  of  GPS  to  a  maximizer  may  only  occur  in  a  finite  number  of  iterations,  and  when 
/  is  locally  constant  along  all  feasible  poll  directions.  On  the  other  hand,  any  reasonable  implementation  of 
MADS  will  generate  more  directions  (a  dense  set  in  the  limit),  which  is  all  that  is  needed  to  avoid  stalling  at 
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the  maximizer.  This  is  consistent  with  Theorem  4.8  which  ensures  that  stalling  at  a  local  maximizer  is  not 
possible  for  MADS,  provided  the  hypotheses  are  met. 


5.2.  An  example  where  GPS  converges  in  an  infinite  number  of  iterations  to  a  saddle  point.  Con¬ 
sider  the  unconstrained  quadratic  optimization  problem  in  which  the  polynomial  objective  function  in  M2  is 
f(a,b)  =  a2  +  3 ab  +  b2.  The  point  (0,0)  is  a  saddle  point,  at  which  the  descent  directions  lie  in  the  cone 
generated  by  a  =  \b{— 3  ±  \/5). 

We  apply  an  instance  of  GPS  where  Dt  =  D  =  {ei,e2,—ei,  —  e2}  is  constant  throughout  all  iterations. 
On  iterations  that  fail  to  improve  the  incumbent,  the  mesh  size  parameter  is  divided  by  16.  On  successful 
iterations  that  follow  an  unsuccessful  one,  the  mesh  size  is  kept  constant,  and  on  other  successful  iterations, 
the  mesh  size  parameter  is  multiplied  by  8.  Thus,  the  GPS  parameters  are  G  =  /  (the  identity  matrix), 
Z  =  D  =  [/;  —  I],  X  =  2,  w~  =  —4  and  w+  =  3. 

Furthermore,  we  use  an  empty  SEARCH  and  an  opportunist  POLL,  i.e.,  an  iteration  terminates  as  soon  as 
an  improved  mesh  point  is  generated.  Moreover,  when  the  iteration  number  k  modulo  3  is  1,  the  POLL  step 
first  evaluates  x*  —  A/. <22,  and  otherwise,  the  POLL  step  first  evaluates  x/.  —  A^ei.  The  order  in  which  the  other 
poll  points  are  explored  is  irrelevant  to  this  example. 

The  initial  parameters  are  xj  =  (1,1)  with  f(x 0)  =  5  and  Ao  =  8.  Figure [5T] displays  the  first  iterates 
generated  by  the  algorithm.  The  figure  also  displays  some  level  sets  of  /. 


We  next  show  that  the  entire  sequence  of  iterates  converges  to  the  origin.  This  happens  because  this 
instance  of  GPS  never  generates  any  trial  points  in  the  cone  where  /  is  negative.  It  either  jumps  over  the 
cone,  which  results  in  an  unsuccessful  iteration,  or  take  a  small  step  which  falls  short  of  reaching  the  cone. 
For  example,  at  iteration  k  =  9,  the  trial  poll  points  are  (|,1),(1,|),(—  g ,  1 )  and  ( 1 ,  —  g ) .  These  four  trial 
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points  are  represented  by  the  symbol  ®  in  the  figure. 

PROPOSITION  5.1.  For  any  integer  i  >  0,  the  GPS  iterates  are  such  that  x^e  =  x3|+i  =  (2~[ ,2“^), 
xie+2  =  (2~{,2^1),  and  A3f  =  2 3~e,  A3^+i  =  A3<?+2  =  2~l~1. 

Proof.  The  proof  is  done  by  induction.  The  result  is  true  for  the  initial  iteration  k  =  0.  Suppose  that 
iteration  k  =  31  is  initiated  with  A*  =  23-/  and  Xk  =  (2~e,2~e).  The  current  objective  function  value  is 
f(xk)  =  5  x  4-^.  Table  5.1  details  the  objective  function  values  at  the  poll  points  for  iterations  k  =  31, 3£  +  1 


and  31  +  2.  Trial  points  that  improve  the  incumbent  appear  in  shaded  boxes.  This  table  shows  that  the  iterate 


k 

Xk 

A  k 

Trial  poll  points  at  Xk  =  ( ak,bk ) 

23-l 

f(al  +  Ak,h)=f(9x2-l,2-l)  =  l09x4-l=f(2-t,9x2-‘))!=f(at,bk  +  Ak) 
f(ak-Ak,bk)  =  /(-7  x  2-',2-<)  =  29  x  4-’-  =f(2-‘,-l  x  2~1))  =f{ak,bt  +  Ak) 

(2-f,2~l) 

2-i-, 

f(ak ,bk-Ak)=f(2~t,  2-‘-'  )  =  1 1  x  4-' - 1 

lt  +  2 

(2-t,2~1-1) 

2-t-i 

f(ak -Al,y=/(2-'-112-'-')  =  5x4-i-1 

3(^+1) 

(2-l-1,2~1-1) 

22-< 

Table  5.1 

Iterations  k  =  3/:  to  k  =  3(7:  -I  I ) 


for  k  =  3(£+  1)  is  (2~1^1 ,2~(~1)  and  that  the  corresponding  mesh  size  parameter  is  22~' .  This  concludes 
the  proof.  ■ 

The  previous  proposition  shows  that  the  entire  sequence  of  iterates  generated  by  GPS  converges  to  the 


saddle  point  (0,0),  which  is  not  a  local  minimizer.  Theorem  4.8  ensures  that  any  MADS  instance  with  an 
asymptotically  dense  set  of  refining  directions  will  not  converge  to  that  saddle  point,  since  the  necessary 
optimality  condition  is  not  satisfied:  vT  =  (—1, 1)  is  a  feasible  direction  for  which  vrV/( 0,0)  =  0  but 


v1  V“/(0,0)v  =  v 


2  3 

3  2 


=  -2 


is  negative. 


5.3.  An  example  where  GPS  reaches  and  stalls  at  a  saddle  point.  Consider  the  bound  constrained 
problem, 

min  f{a,b)  =  99a2—20ab  +  b2  =  (9a  —  b)(lla  —  b). 

—2<a,b<2 

At  the  saddle  point  (0,0),  directions  of  descent  lie  only  in  the  narrow  cone  formed  by  the  lines  b  =  9a  and 
b  =  lla.  Thus  to  avoid  stalling  at  the  saddle  point,  GPS  or  MADS  would  have  to  generate  a  feasible  iterate 
that  lies  inside  this  cone  (see  Figure |5T2)>.  In  this  example,  the  SEARCH  step  is  empty  and  the  initial  point  is 
chosen  to  be  (1.01,  0.93).  This  starting  point  is  chosen  to  be  non-integral  to  make  it  more  difficult  for  GPS 
to  reach  the  integral  point  (0,0).  Both  GPS  and  MADS  were  run  using  the  NOMAD  software  package  ifDTl 
with  primarily  default  settings. 

GPS  (using  the  standard  coordinate  directions  and  their  negatives  as  poll  directions)  reaches  the  saddle 
point  at  the  358th  function  evaluation  with  a  poll  size  parameter  of  10  1 7 .  This  implies  that,  regardless  of 
the  termination  tolerance  chosen,  it  stalls  there  because  none  of  the  poll  directions  are  directions  of  descent. 
On  the  other  hand,  NOMAD’s  implementation  of  LTMADS  successfully  moved  off  of  the  saddle  point  to 
reach  a  local  minimizer  in  100  of  100  runs.  This  is  again  consistent  with  Theorem  4.8  since  vT  =  (1, 10)  is 
a  feasible  direction  for  which  v;7V/( 0,0)  =  0  but 


v1  V"/(0,0)v  =  v1 


198  -20 

-20  2 


=  -2 
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Cones  of  Negative  Curvature 


FIG.  5.2.  For  f(a,b)  —  (9  a  —  b)(\\a  —  b),  the  cones  of  descent  at  the  saddle  point  (0,0)  are  shown  in  the  shaded  area  between 
the  lines  b  —  9a  and  b  —  11  a. 


is  negative. 


6.  Concluding  Remarks.  The  theoretical  results  presented  here  establish  strong  convergence  results 
for  MADS.  In  spite  of  MADS  being  a  derivative-free  method,  we  have  shown  convergence  of  a  subsequence 
of  MADS  iterates  to  a  second-order  stationary  point  under  conditions  weaker  than  standard  Newton  assump¬ 
tions;  namely,  that  /  is  continuously  differentiable  with  Lipschitz  derivatives  near  the  limit  point.  Moreover, 
if  Cl  is  locally  convex  and  /  twice  strictly  differentiable  near  the  limit  point,  then  the  limit  point  is  a  local 
minimizer  for  m 

In  Section  [5]  we  provided  examples  to  illustrate  the  superior  convergence  properties  of  MADS  over 
GPS.  However,  since  our  implementation  involves  random  selection  of  positive  spanning  directions,  the 


convergence  properties  established  in  Section  4.3  are  achieved,  in  practice,  with  probability  one.  We  envi¬ 


sion  a  future  area  of  research  being  the  clever  enumeration  of  these  directions  so  that  the  stronger  type  of 
convergence  is  retained  by  an  implementable  instance  of  the  algorithm.  Specifically,  we  would  like  to  deter¬ 
ministically  generate  an  asymptotically  dense  set  of  directions  in  such  a  way  that,  after  any  finite  number  of 
iterations,  the  directions  used  by  the  algorithm  are  uniformly  spaced  (or  as  close  to  it  as  possible). 
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